feat(validators): support check-failing-tests in data-paths mode + --update-expected#84
Merged
Merged
Conversation
…update-expected
Enable check-failing-tests to run against externally-supplied negative fixtures
in Data Path Mode (previously Catalog/Domain Mode only), and add an
--update-expected flag to (re)record each .expected snapshot from the live
validation report.
- registry_resolver.create_temporary_domain: classify each --data-paths file by
its parent directory name -- files under invalid/ register as invalid
test-data (matching OMB's tests/data/{domain}/{valid,invalid}/ convention),
all others as valid. Temporary test ids now include the test_type so the two
groups never collide on a catalog key.
- validation_suite: relax the data-paths guard so only check-artifact-coherence
stays unsupported; thread update_expected through check_failing_tests_all (on
a 210 it writes the boxed report to <fixture>.expected, else compares as
before). Default-off: zero behaviour change when the flag is unset.
…aa7702c The committed openlabel-v2 artifacts predate gen-owl/gen-shacl changes now present in the pinned fork (feat/envited-x-pipeline @ 6aa7702c). CI's make generate + "verify no changes", and the generate-linkml pre-commit hook, regenerate them and fail on the diff. Refresh so committed == generated: - SHACL: boolean flag comparison str(?flag) != "true" -> ?flag != true (compare against xsd:boolean rather than a string); templated sh:message no longer carries an @en tag; deterministic constraint re-ordering. - OWL: enum range emitted as owl:oneOf ( "EdgeNone" ) instead of a bare xsd:string restriction. No schema or tooling change; make generate is now a no-op. This drift is pre-existing on main (unrelated to the check-failing-tests tooling on this branch); refreshing here unblocks the Generate Artifacts and Standards & Syntax checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Two small, additive, default-off improvements to
validation_suiteso that adownstream repository can validate its OWN JSON-LD instances (positive and
negative) against an external artifact bundle, without duplicating OMB's runner,
report formatter or snapshot logic.
1.
check-failing-testsnow works in Data Path ModePreviously
check-failing-testswas Catalog/Domain Mode only. It now also runsagainst externally-supplied fixtures passed via
--data-paths:create_temporary_domainclassifies each file by the name of its parentdirectory -- files under an
invalid/directory are registered asinvalidtest-data (matching OMB's own
tests/data/{domain}/{valid,invalid}/convention),everything else as
valid. Temporary test ids now include thetest_typesothe valid/invalid groups can never collide on a catalog key.
Only
check-artifact-coherenceremains unsupported in data-paths mode (it needsthe full catalog structure).
2.
--update-expectedsnapshot recordercheck-failing-tests --update-expected(re)records each negative test's.expectedsnapshot from the live boxed validation report instead of comparing.Intended for use after an intentional schema or OMB/pyshacl change, then review the
diff. When the flag is unset, behaviour is byte-for-byte unchanged.
Behaviour note
A
check-data-conformancerun over files that live under a directory literallynamed
invalidwill now treat them as negative fixtures (skipped as conformancetargets) rather than positive ones. This matches the repository's
{valid,invalid}convention and only affects that exact directory name.
Testing
ruff check+ruff formatclean.pytest tests/unit tests/integration-> 471 passed,1 xfailed (no regressions; the new paths are exercised by a downstream consumer).